PRCV 2025 Generalized Few-Shot Semantic Segmentation Feature Decoupling Prototype Learning Joint Learning

Generalized Few-Shot Semantic Segmentation Based on Relevant Intrinsic Feature Enhancement

Lulu Jiang1   Yaozheng Xia1   Shaorong Wang1,2

1 School of Information Science and Technology, School of Artificial Intelligence, Beijing Forestry University, Beijing, China
2 Engineering Research Center for Forestry-oriented Intelligent Information Processing of National Forestry and Grassland Administration, Beijing, China

Paper

TL;DRPropose RIFE method that addresses intra-class variations in generalized few-shot semantic segmentation through feature decoupling/recombination and joint learning, achieving state-of-the-art on PASCAL-5i and COCO-20i

Abstract

RIFE tackles intra-class variations in generalized few-shot semantic segmentation by proposing a feature decoupling and recombination mechanism that mines robust intrinsic features, and a joint learning strategy that concurrently segments both support and query images during novel class inference to verify prototype discriminability.

Method

Existing prototype-based methods suffer from intra-class variations between support and query images that degrade prototype effectiveness.

Decouple foreground/background features across support-query branches via self-attention and cross-attention mechanisms, recombine them to reduce intra-class variations while enhancing inter-class discrimination, and validate prototypes by jointly segmenting both support and query images during inference.

Feature Decoupling

Feature Decoupling and Recombination Mechanism

Separates support features into foreground/background via annotation masks, drives query feature decoupling through cross-branch foreground interaction without query mask supervision, then uses self-attention and cross-attention to enhance intra-branch discrimination and inter-branch semantic alignment

Joint Learning

Joint Support-Query Segmentation Strategy

Concurrently predicts segmentation masks for both support and query images during novel class inference, using support predictions to validate prototype discriminability and jointly optimize both branches for more accurate and stable prototype representations

Ensemble Module

Base-Novel Class Ensemble Fusion

Fuses base-class and novel-class prediction branches using a threshold-based strategy: novel class predictions take priority when confidence exceeds threshold, otherwise base class predictions are used, enabling seamless integration of both segmentation outputs

Fig. 1. The model structure of RIFE.

Fig. 1. The model structure of RIFE.

RIFE comprises three core components: Base Prediction (standard supervised learning on base classes via PSPNet with ResNet-50 backbone), Novel Prediction (meta-learning with feature decoupling, prototype generation, and joint learning), and Ensemble Module (threshold-based fusion of base and novel predictions).

Fig. 2. Diagram of foreground self-attention within a branch.

Fig. 2. Diagram of foreground self-attention within a branch.

Self-attention within each branch refines foreground features by computing Q/K/V projections and applying scaled dot-product attention with residual connections, enhancing discriminability of task-relevant features in both support and query branches.

Fig. 3. Diagram of foreground cross-attention between branches.

Fig. 3. Diagram of foreground cross-attention between branches.

Cross-attention between branches aligns support and query foreground features by computing attention weights from support features to query features, reducing intra-class support-query discrepancies while promoting foreground-background decoupling across branches.

Fig. 4. Diagram of background feature modeling based on Vision Transformer.

Fig. 4. Diagram of background feature modeling based on Vision Transformer.

Results

PASCAL-5i (5-shot)
71.00%
Total mIoU
+0.72% over POP (70.28%), SOTA Best overall mIoU on PASCAL-5i 5-shot setting
PASCAL-5i (1-shot)
42.65%
Novel mIoU
+1.41% over BCM (41.24%), SOTA Best novel-class mIoU in 1-shot setting
COCO-20i (5-shot)
48.89%
Total mIoU
+0.14% over POP (48.75%), SOTA Best overall mIoU on COCO-20i 5-shot setting
COCO-20i (5-shot)
30.95%
Novel mIoU
+0.35% over BCM (30.60%), SOTA Best novel-class mIoU on COCO-20i 5-shot
MethodPASCAL-5i 1s NovelPASCAL-5i 5s NovelCOCO-20i 1s NovelCOCO-20i 5s NovelPASCAL-5i 5s Total
CAPL18.8522.417.0511.0555.72
PKL26.9034.4011.0414.9161.18
DIaM35.1155.3117.2228.7366.97
POP35.5155.8715.3129.9770.28
BCM41.2455.3618.2830.6067.45
RIFE (Ours) ours42.6555.9516.9830.9571.00

RIFE achieves state-of-the-art performance on both PASCAL-5i and COCO-20i benchmarks across 1-shot and 5-shot settings. The feature decoupling and recombination mechanism (method c) outperforms simpler variants (methods a, b) by +3.16% Total mIoU on PASCAL-5i 5-shot. Joint learning strategy (strategy b) adds +1.23% Total mIoU over query-only optimization.

Qualitative Results

Fig. 5. The comparison of prediction results between our method and the CAPL method under the 5 -shot setting on the PASCAL $- {5}^{i}$ dataset.

Fig. 5. The comparison of prediction results between our method and the CAPL method under the 5 -shot setting on the PASCAL $- {5}^{i}$ dataset.

Takeaway

Cross-branch feature decoupling and recombination is an effective strategy for reducing intra-class variations in few-shot segmentation. Joint learning that validates prototypes by concurrently segmenting both support and query images improves prototype discriminability and segmentation accuracy without requiring additional annotations.

Citation

@inproceedings{jiang2025generalized,
  title={Generalized Few-Shot Semantic Segmentation Based on Relevant Intrinsic Feature Enhancement},
  author={Jiang, Lulu and Xia, Yaozheng and Wang, Shaorong},
  booktitle={Pattern Recognition and Computer Vision (PRCV)},
  pages={234--247},
  year={2025},
  publisher={Springer},
  doi={10.1007/978-981-95-4987-0_17}
}